A Process-Oriented Heuristic for Model Selection

Current methods to avoid overrtting are either data-oriented (using separate data for validation) or representation-oriented (penalizing complexity in the model). This paper proposes process-oriented evaluation, where a model's expected generalization error is computed as a function of the search process that led to it. The paper develops the necessary theoretical framework, and applies it to one type of learning: rule induction. A process-oriented version of the CN2 rule learner is empirically compared with the default CN2. The process-oriented version is more accurate in a large majority of the datasets, with high signiicance, and also produces simpler models. Experiments in artiicial domains suggest that process-oriented evaluation is particularly useful in high-dimensional domains.